To Train or Not To Train: Optimal Treatment Assignment Rules Using Welfare-to-Work Experiments

Planners often face the especially difficult and important task of assigning programs or treatments to optimize outcomes. Using the recent welfare-to-work reforms as an illustration, this paper considers the normative problem of how administrators might use data from randomized experiments to assign treatments. Under the new welfare system, state governments must design welfare programs to optimize employment. With experimental results in-hand, planners observe the average effect of training on employment but may not observe the individual effect of training. If the effect of a treatment varies across individuals, the planner faces a decision problem under ambiguity (Manski, 1998). In this setting, I find a straightforward proposition formalizes conditions under which a planner should reject particular decision rules as being inferior. An optimal decision rule, however, may not be revealed. In the absence of strong assumptions about the degree of heterogeneity in the population or the information known by the planner, the data are inconclusive about the efficacy of most assignment rules.ambiguity, randomized experiments, treatment choice, welfare-to-work programs

Identification of Expected Outcomes in a Data Error Mixing Model with Multiplicative Mean Independence

We consider the problem of identifying a mean outcome in corrupt sampling where the observed outcome is a mixture of the distribution of interest and some other distribution. We make two contributions to this literature. First, the statistical independence assumption maintained under contaminated sampling is relaxed to the weaker assumption that the outcome is mean independent of the mixing process. We then generalize this restriction to allow the two conditional means to differ by a known or bounded factor of proportionality. Second, in the special case of a binary outcome, we consider the possibility that draws from the alternative distribution are known to be erroneous, as might be the case in a mixture model of response error. We illustrate how these assumptions can be used to inform researchers about the population's use of illicit drugs in the presence of nonrandom reporting errors. In this application, we find that a response error model with multiplicative mean independence is easy to motivate and can have substantial identifying power.

Disability and Employment: Reevaluating the Evidence in Light of Reporting Errors

Measurement error in health and disability status has been widely accepted as a central problem for social science research. Long-standing debates about the prevalence of disability, the role of health in labor market outcomes, and the influence of federal disability policy on declining employment rates have all emphasized issues regarding the reliability of self-reported disability. In addition to random error, inaccuracy in survey datasets may be produced by a host of economic, social, and psychological factors that can lead respondents to misreport work capacity. We develop a nonparametric foundation for assessing how assumptions on the reporting error process affect inferences on the employment gap between the disabled and nondisabled. Rather than imposing the strong assumptions required to obtain point identification, we derive sets of bounds that formalize the identifying power of primitive nonparametric assumptions that appear to share broad consensus in the literature. Within this framework, we introduce a finite-sample correction for the analog estimator of the monotone instrumental variable (MIV) bound. Our empirical results suggest that conclusions derived from conventional latent variable reporting error models may be driven largely by ad hoc distributional and functional form restrictions. Under relatively weak nonparametric assumptions, nonworkers appear to systematically overreport disability.

Inferring Disability Status from Corrupt Data

In light of widespread concerns about the reliability of self-reported disability, we investigate what can be learned about the prevalence of work disability under various assumptions on the reporting error process. Developing a nonparametric bounding framework, we provide tight inferences under our strongest assumptions but then find that identification deteriorates rapidly as the assumptions are relaxed. For example, we find that inferences are highly sensitive to how one models potential inconsistencies between subjective self-assessments of work limitation and more objective measures of functional limitation. These two indicators appear to measure markedly different aspects of health status.

Monotone Instrumental Variables: With an Application to the Returns to Schooling

Econometric analyses of treatment response commonly use instrumental variable (IV) assumptions to identify treatment effects. Yet the credibility of IV assumptions is often a matter of considerable disagreement. There is therefore good reason to consider weaker but more credible assumptions. To this end, we introduce monotone instrumental variable (MIV) assumptions and the important special case of monotone treatment selection (MTS). We study the identifying power of MIV assumptions alone and combined with the assumption of monotone treatment response (MTR). We present an empirical application using the MTS and MTR assumptions to place upper bounds on the returns to schooling.

What Do Welfare-to-Work Demonstrations Reveal to Welfare Reformers?

Under the new welfare system, states must design and institute programs that both provide assistance and encourage work, two objectives that have thus far appeared incompatible. Will states meet these new requirements? For many innovative programs, the randomized welfare-to-work experiments conducted over the last three decades may be the only source of observed data. While these experiments yield information on the outcomes of mandated treatments, the new regime permits states and localities much discretion. Using data from four experiments conducted in the mid-1980s, this study examines what welfare-to-work demonstrations reveal about outcomes when the treatments are heterogenous. In the absence of assumptions, these data allow us to draw only limited inferences about the labor market outcomes of welfare recipients. Combined with prior information, however, data from experimental demonstrations are informative, suggesting either that the long run federal requirements cannot be met or that these standards will only be met under special circumstances.

Using Performance Standards to Evaluate Social Programs with Incomplete Outcome Data: General Issues and Application to a Higher Education Block Grant Program

The basic idea of program evaluation is both simple and appealing. Program outcomes are measured and compared to some minimum performance standard or threshold. In practice, however, evaluation is quite difficult. Two fundamental problems of outcome measurement must be addressed. The first, which we call the problem of auxiliary outcomes, is that we do not observe outcome of interest. The second, which we call the problem of counterfactual outcomes, is that we do not observe the threshold standard. This paper examines how performance standards should be set and applied in the face of these problems in measuring outcomes. In particular, we consider the problem of evaluating the new World Bank sponsored Quality of Undergraduate Education (QUE) program. This competitive block grant program is to be judged by the program's effects on student outcomes, not by the particular ways in which the grantee departments use their funds. Our central message is that the proper way to implement standards varies with the prior information that the evaluator can credibly bring to bear to compensate for incomplete outcome data. An evaluator, confronted with the auxiliary and counterfactual outcomes problems, should combine the available data with credible assumptions on treatments and outcomes. Given this information, the performance of a program may be deemed acceptable, unacceptable or indeterminate.

Identification of Expected Outcomes in a Data Error Mixing Model With Multiplicative Mean Independence

We consider the problem of identifying a mean outcome in corrupt sampling where the observed outcome is drawn from a mixture of the distribution of interest and another distribution. Relaxing the contaminated sampling assumption that the outcome is statistically independent of the mixing process, we assess the identifying power of an assumption that the conditional means of the distributions differ by a factor of proportionality. For binary outcomes, we consider the special case that all draws from the alternative distribution are erroneous. We illustrate how these models can inform researchers about illicit drug use in the presence of reporting errors

Disability and Employment: Reevaluating the Evidence in Light of Reporting Errors

Measurement error in health and disability status has been widely accepted as a central problem in social science research. Long-standing debates about the prevalence of disability, the role of health in labor market outcomes, and the influence of federal disability policy on declining employment rates have all emphasized issues regarding the reliability of self-reported disability. In addition to random error, inaccuracy in survey datasets may be produced by a host of economic, social, and psychological factors that can lead respondents to misreport work capacity. We develop a nonparametric foundation for assessing how assumptions on the reporting error process affect inferences on the employment gap between the disabled and nondisabled. Rather than imposing the strong assumptions required to obtain point identification, we derive sets of bounds that formalize the identifying power of primitive nonparametric assumptions that appear to share broad consensus in the literature. Within this framework, we introduce a finite-sample correction for the analog estimator of the monotone instrumental variable (MIV) bound. Our empirical results suggest that conclusions derived from conventional latent variable reporting error models may be driven largely by ad hoc distributional and functional form restrictions. We also find that under relatively weak nonparametric assumptions, nonworkers appear to systematically overreport disability